This invention is in the field of interactive display systems. Embodiments of this invention are more specifically directed to the positioning of the location at a display to which a control device is pointing during the interactive operation of a computer system.
The ability of a speaker to communicate a message to an audience is generally enhanced by the use of visual information, in combination with the spoken word. In the modern era, the use of computers and associated display systems to generate and display visual information to audiences has become commonplace, for example by way of applications such as the POWERPOINT presentation software program available from Microsoft Corporation. For large audiences, such as in an auditorium environment, the display system is generally a projection system (either front or rear projection). For smaller audiences such as in a conference room or classroom environment, flat-panel (e.g., liquid crystal) displays have become popular, especially as the cost of these displays has fallen over recent years. New display technologies, such as small projectors (“pico-projectors”), which do not require a special screen and thus are even more readily deployed, are now reaching the market. For presentations to very small audiences (e.g., one or two people), the graphics display of a laptop computer may suffice to present the visual information. In any case, the combination of increasing computer power and better and larger displays, all at less cost, has increased the use of computer-based presentation systems, in a wide array of contexts (e.g., business, educational, legal, entertainment).
A typical computer-based presentation involves the speaker standing remotely from the display system, so as not to block the audience's view of the visual information. Because the visual presentation is computer-generated and computer-controlled, the presentation is capable of being interactively controlled, to allow selection of visual content of particular importance to a specific audience, annotation or illustration of the visual information by the speaker during the presentation, and invocation of effects such as zooming, selecting links to information elsewhere in the presentation (or online), moving display elements from one display location to another, and the like. This interactivity greatly enhances the presentation, making it more interesting and engaging to the audience.
The ability of a speaker to interact, from a distance, with displayed visual content, is therefore desirable. More specifically, a hand-held device that a remotely-positioned operator could use to point to, and interact with, the displayed visual information is therefore desirable.
U.S. Pat. No. 8,217,997, issued Jul. 10, 2012, entitled “Interactive Display System”, commonly assigned herewith and incorporated herein by reference, describes an interactive display system including a wireless human interface device (“HID”) constructed as a handheld pointing device including a camera or other video capture system. The pointing device captures images displayed by the computer, including one or more human-imperceptible positioning targets inserted by the computer into the displayed image data. The location, size, and orientation of the recovered positioning target identify the aiming point of the remote pointing device relative to the display. Temporal sequencing of the positioning targets (either human-perceptible or human-imperceptible) to position the pointing device is also described.
The positioning of the aiming point of the pointing device according to the approach described in the above-referenced U.S. Pat. No. 8,217,997 is performed at a rate corresponding to the frame rate of the display system. More specifically, a new position can be determined as each new frame of data is displayed, by the combination of the new frame (and its positioning target) and the immediately previous frame (and its complementary positioning target). This approach works quite well in many situations, particularly in the context of navigating and controlling a graphical user interface in a computer system, such as pointing to and “clicking” icons, click-and-drag operations involving displayed windows and frames, and the like. A particular benefit of this approach described in U.S. Pat. No. 8,217,997, is that the positioning is “absolute”, in the sense that the result of the determination is a specific position on the display (e.g., pixel coordinates). The accuracy of the positioning carried out according to this approach is quite accurate over a wide range of distances between the display and the handheld device, for example ranging from in physical contact with the display screen to tens of feet away.
But because of the dependence of the positioning rate on the display frame rate, the ability of this approach has limitations. Rapid movement of the handheld device, for example while “writing” on the display screen in an electronic interactive “white board” application, can present motion that may not be fully captured by positioning at the frame rate. In addition, time lag between movement of the handheld device and the display response can be noticeable to the user and the audience in some situations.
Conventional human interface devices based on motion sensors are known in the art. Motion sensors sense motion of the device over time, for example between sample times. Examples of motion sensors include inertial sensors such as accelerometers, gyroscopes, magnetic field sensors such as magnetometers, and visual systems such as those used in optical mice. The positioning result based on motion sensors is relative, in the sense that an absolute position of the display is not directly determined, but rather the motion sensors determine the pointed-to location relative to that at a previous point in time. However, the sample rate at which motion sensor-based pointing devices operate is not limited by the frame rate of the display, and can be much higher, assuming proper registration of the relative positioning. In addition, fewer computations are required to derive the relative positioning result, as compared with those required for absolute positioning. Unfortunately, however, because the positioning provided by these devices is relative, drift or other error can accumulate over time. Error is exacerbated for those devices relying on accelerometer motion sensing, as two integrations are required in order to convert sensed accelerations into linear distances. As such, the accuracy of relative positioning based on motion sensors is generally inferior to that of absolute positioning approaches.
To summarize, both absolute and relative systems for determining the position of a display at which a human interface device is pointing are known in the art. Absolute positioning provides good accuracy but at a relatively slow maximum rate, while relative positioning can operate at a high rate, but is vulnerable to error.